I gave up on vests about 15 years ago. It was just to much. What I did instead was start using a . . . well, you can be kind and call it a messenger bag, or you can call it what it is: a purse.
It holds a box call, a couple-three pot calls. A dozen mouth calls ammo and extras. I throw the rest of the kit in a musette bag. I attach a lightweight foam seat to the back of the purse for my butt.
If you're looking for a good reason for not wearing a vest, it comes down to two things:
1) Covering the back door. The easiest way to deal with a gobbler coming up on your six is to roll to one side of the tree or the other and go prone. With a vest, I found all my gear falling out or clanking.
2) Going light and fast. Some times I will shed all my gear and do an end-around with just my shotgun and a mouth call. It is very easy to dump the load when it falls off at just the touch of a buckle.
Here's a picture of the whole rig in action. I was trying to get out before a big thunderstorm hit and rolled up all my outerwear into the musette bag. The wind was up, so the lightweight pads were blowing. Normally the musette bag is nearly empty, but I like the extra room for storing my sweater, etc on the hike out.